Pandas Profiling Report

Dataset statistics

Number of variables	11
Number of observations	987269
Missing cells	27277
Missing cells (%)	0.3%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	82.9 MiB
Average record size in memory	88.0 B

Variable types

NUM	8
BOOL	2
CAT	1

Warnings

`row_id` is highly correlated with `Unnamed: 0` and 1 other fields	High correlation
`Unnamed: 0` is highly correlated with `row_id` and 1 other fields	High correlation
`user_id` is highly correlated with `Unnamed: 0` and 1 other fields	High correlation
`prior_question_elapsed_time` has 23356 (2.4%) missing values	Missing
`Unnamed: 0` has unique values	Unique
`row_id` has unique values	Unique
`user_answer` has 274421 (27.8%) zeros	Zeros

Reproduction

Analysis started	2020-12-08 09:47:33.673751
Analysis finished	2020-12-08 09:49:18.367265
Duration	1 minute and 44.69 seconds
Software version	pandas-profiling v2.9.0
Download configuration	config.yaml

Unnamed: 0
Real number (ℝ_≥0)

HIGH CORRELATION
UNIQUE

Distinct	987269
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	51061759.65
Minimum	78225
Maximum	101224061
Zeros	0
Zeros (%)	0.0%
Memory size	7.5 MiB

Quantile statistics

Minimum	78225
5-th percentile	6486992.4
Q1	26890846
median	50998218
Q3	75022565
95-th percentile	97041437.6
Maximum	101224061
Range	101145836
Interquartile range (IQR)	48131719

Descriptive statistics

Standard deviation	28483682.82
Coefficient of variation (CV)	0.5578280697
Kurtosis	-1.155367124
Mean	51061759.65
Median Absolute Deviation (MAD)	24107039
Skewness	0.01417549326
Sum	5.041169239e+13
Variance	8.113201871e+14
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
18787089	1	< 0.1%
78014865	1	< 0.1%
96504141	1	< 0.1%
41976140	1	< 0.1%
37794122	1	< 0.1%
92315976	1	< 0.1%
9541752	1	< 0.1%
26483807	1	< 0.1%
2708973	1	< 0.1%
33612096	1	< 0.1%
Other values (987259)	987259	> 99.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
78225	1	< 0.1%
78226	1	< 0.1%
78227	1	< 0.1%
78228	1	< 0.1%
78229	1	< 0.1%

Value	Count	Frequency (%)
101224061	1	< 0.1%
101224060	1	< 0.1%
101224059	1	< 0.1%
101224058	1	< 0.1%
101224057	1	< 0.1%

row_id
Real number (ℝ_≥0)

HIGH CORRELATION
UNIQUE

Distinct	987269
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	51061759.65
Minimum	78225
Maximum	101224061
Zeros	0
Zeros (%)	0.0%
Memory size	7.5 MiB

Quantile statistics

Minimum	78225
5-th percentile	6486992.4
Q1	26890846
median	50998218
Q3	75022565
95-th percentile	97041437.6
Maximum	101224061
Range	101145836
Interquartile range (IQR)	48131719

Descriptive statistics

Standard deviation	28483682.82
Coefficient of variation (CV)	0.5578280697
Kurtosis	-1.155367124
Mean	51061759.65
Median Absolute Deviation (MAD)	24107039
Skewness	0.01417549326
Sum	5.041169239e+13
Variance	8.113201871e+14
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
18787089	1	< 0.1%
78014865	1	< 0.1%
96504141	1	< 0.1%
41976140	1	< 0.1%
37794122	1	< 0.1%
92315976	1	< 0.1%
9541752	1	< 0.1%
26483807	1	< 0.1%
2708973	1	< 0.1%
33612096	1	< 0.1%
Other values (987259)	987259	> 99.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
78225	1	< 0.1%
78226	1	< 0.1%
78227	1	< 0.1%
78228	1	< 0.1%
78229	1	< 0.1%

Value	Count	Frequency (%)
101224061	1	< 0.1%
101224060	1	< 0.1%
101224059	1	< 0.1%
101224058	1	< 0.1%
101224057	1	< 0.1%

timestamp
Real number (ℝ_≥0)

Distinct	761364
Distinct (%)	77.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	7555878183
Minimum	0
Maximum	8.077425916e+10
Zeros	3969
Zeros (%)	0.4%
Memory size	7.5 MiB

Quantile statistics

Minimum	0
5-th percentile	471140.2
Q1	462058218
median	2305517139
Q3	9166741308
95-th percentile	3.386438115e+10
Maximum	8.077425916e+10
Range	8.077425916e+10
Interquartile range (IQR)	8704683090

Descriptive statistics

Standard deviation	1.217767912e+10
Coefficient of variation (CV)	1.611682829
Kurtosis	8.48916776
Mean	7555878183
Median Absolute Deviation (MAD)	2221170700
Skewness	2.682689923
Sum	7.459684298e+15
Variance	1.482958688e+20
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	3969	0.4%
219754	9	< 0.1%
310705	8	< 0.1%
1028294	8	< 0.1%
321106	8	< 0.1%
886108	8	< 0.1%
284907	7	< 0.1%
473607	7	< 0.1%
141453	7	< 0.1%
407171	7	< 0.1%
Other values (761354)	983231	99.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	3969	0.4%
2127	1	< 0.1%
3480	1	< 0.1%
3485	1	< 0.1%
3994	1	< 0.1%

Value	Count	Frequency (%)
8.077425916e+10	1	< 0.1%
8.077422286e+10	1	< 0.1%
8.077418077e+10	1	< 0.1%
8.077415011e+10	1	< 0.1%
8.077412023e+10	1	< 0.1%

user_id
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	3935
Distinct (%)	0.4%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1086222355
Minimum	1710599
Maximum	2147379374
Zeros	0
Zeros (%)	0.0%
Memory size	7.5 MiB

Quantile statistics

Minimum	1710599
5-th percentile	141078962
Q1	574033464
median	1080358589
Q3	1596427076
95-th percentile	2058869515
Maximum	2147379374
Range	2145668775
Interquartile range (IQR)	1022393612

Descriptive statistics

Standard deviation	604002707.4
Coefficient of variation (CV)	0.5560580713
Kurtosis	-1.155781392
Mean	1086222355
Median Absolute Deviation (MAD)	508556150
Skewness	0.01532202043
Sum	1.072393658e+15
Variance	3.648192706e+17
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
1641944791	12146	1.2%
1816238574	8209	0.8%
346059871	8170	0.8%
814836243	7950	0.8%
1217765167	7792	0.8%
586151578	7101	0.7%
1416959515	6808	0.7%
1430438958	6430	0.7%
847874849	6032	0.6%
1357086301	5759	0.6%
Other values (3925)	910872	92.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1710599	22	< 0.1%
1906069	30	< 0.1%
2101969	253	< 0.1%
2198581	28	< 0.1%
2381110	62	< 0.1%

Value	Count	Frequency (%)
2147379374	41	< 0.1%
2145920530	611	0.1%
2145381578	30	< 0.1%
2145343445	107	< 0.1%
2145253714	1082	0.1%

content_id
Real number (ℝ_≥0)

Distinct	13310
Distinct (%)	1.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5222.613857
Minimum	0
Maximum	32736
Zeros	79
Zeros (%)	< 0.1%
Memory size	7.5 MiB

Quantile statistics

Minimum	0
5-th percentile	320
Q1	2081
median	5030
Q3	7391
95-th percentile	10688
Maximum	32736
Range	32736
Interquartile range (IQR)	5310

Descriptive statistics

Standard deviation	3852.192066
Coefficient of variation (CV)	0.7375984845
Kurtosis	7.314665309
Mean	5222.613857
Median Absolute Deviation (MAD)	2726
Skewness	1.613867732
Sum	5156124760
Variance	14839383.71
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
6116	2197	0.2%
6173	2105	0.2%
4120	2050	0.2%
175	2019	0.2%
7876	1938	0.2%
7900	1851	0.2%
4492	1787	0.2%
2063	1772	0.2%
2064	1772	0.2%
2065	1772	0.2%
Other values (13300)	968006	98.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	79	< 0.1%
1	70	< 0.1%
2	496	0.1%
3	217	< 0.1%
4	307	< 0.1%

Value	Count	Frequency (%)
32736	84	< 0.1%
32625	74	< 0.1%
32570	26	< 0.1%
32535	42	< 0.1%
32491	15	< 0.1%

content_type_id
Boolean

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	7.5 MiB

0	967834
1	19435

Common Values
Chart

Value	Count	Frequency (%)
0	967834	98.0%
1	19435	2.0%

task_container_id
Real number (ℝ_≥0)

Distinct	9210
Distinct (%)	0.9%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	760.9032736
Minimum	0
Maximum	9210
Zeros	3961
Zeros (%)	0.4%
Memory size	7.5 MiB

Quantile statistics

Minimum	0
5-th percentile	9
Q1	97
median	346
Q3	908
95-th percentile	3058
Maximum	9210
Range	9210
Interquartile range (IQR)	811

Descriptive statistics

Standard deviation	1130.052798
Coefficient of variation (CV)	1.485146453
Kurtosis	11.60782145
Mean	760.9032736
Median Absolute Deviation (MAD)	299
Skewness	3.005098215
Sum	751216214
Variance	1277019.327
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
14	8130	0.8%
15	8080	0.8%
4	6960	0.7%
5	6945	0.7%
6	6932	0.7%
7	6853	0.7%
11	4044	0.4%
10	3988	0.4%
8	3967	0.4%
9	3967	0.4%
Other values (9200)	927403	93.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	3961	0.4%
1	3964	0.4%
2	3957	0.4%
3	3944	0.4%
4	6960	0.7%

Value	Count	Frequency (%)
9210	1	< 0.1%
9209	1	< 0.1%
9208	1	< 0.1%
9207	1	< 0.1%
9206	1	< 0.1%

user_answer
Real number (ℝ)

ZEROS

Distinct	5
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1.376106208
Minimum	-1
Maximum	3
Zeros	274421
Zeros (%)	27.8%
Memory size	7.5 MiB

Quantile statistics

Minimum	-1
5-th percentile	0
Q1	0
median	1
Q3	3
95-th percentile	3
Maximum	3
Range	4
Interquartile range (IQR)	3

Descriptive statistics

Standard deviation	1.192983359
Coefficient of variation (CV)	0.8669268057
Kurtosis	-1.298870727
Mean	1.376106208
Median Absolute Deviation (MAD)	1
Skewness	0.08394766547
Sum	1358587
Variance	1.423209296
Monotocity	Not monotonic

Histogram with fixed size bins (bins=5)

Value	Count	Frequency (%)
0	274421	27.8%
1	262791	26.6%
3	253987	25.7%
2	176635	17.9%
-1	19435	2.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-1	19435	2.0%
0	274421	27.8%
1	262791	26.6%
2	176635	17.9%
3	253987	25.7%

Value	Count	Frequency (%)
3	253987	25.7%
2	176635	17.9%
1	262791	26.6%
0	274421	27.8%
-1	19435	2.0%

answered_correctly
Categorical

Distinct	3
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	7.5 MiB

1	635571
0	332263
-1	19435

Frequencies
Length

Value	Count	Frequency (%)
1	635571	64.4%
0	332263	33.7%
-1	19435	2.0%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	2
Median length	1
Mean length	1.019685618
Min length	1

prior_question_elapsed_time
Real number (ℝ_≥0)

MISSING

Distinct	1687
Distinct (%)	0.2%
Missing	23356
Missing (%)	2.4%
Infinite	0
Infinite (%)	0.0%

Mean	25320.71289
Minimum	0
Maximum	300000
Zeros	1699
Zeros (%)	0.2%
Memory size	7.5 MiB

Quantile statistics

Minimum	0
5-th percentile	6000
Q1	16000
median	21000
Q3	29250
95-th percentile	57000
Maximum	300000
Range	300000
Interquartile range (IQR)	13250

Descriptive statistics

Standard deviation	20814.85445
Coefficient of variation (CV)	0.8220485157
Kurtosis	54.0655943
Mean	25320.71289
Median Absolute Deviation (MAD)	6000
Skewness	5.484981435
Sum	2.440696433e+10
Variance	433258165.8
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
17000	49942	5.1%
16000	46223	4.7%
18000	44479	4.5%
19000	39155	4.0%
15000	35912	3.6%
20000	34993	3.5%
21000	32531	3.3%
22000	30092	3.0%
23000	26301	2.7%
14000	25981	2.6%
Other values (1677)	598304	60.6%
(Missing)	23356	2.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1699	0.2%
200	1	< 0.1%
250	5	< 0.1%
333	1073	0.1%
600	12	< 0.1%

Value	Count	Frequency (%)
300000	1172	0.1%
299000	5	< 0.1%
298000	1	< 0.1%
297000	2	< 0.1%
296000	4	< 0.1%

prior_question_had_explanation
Boolean

Distinct	2
Distinct (%)	< 0.1%
Missing	3921
Missing (%)	0.4%
Memory size	7.5 MiB

True	870275
False	113073
(Missing)	3921

Common Values
Chart

Value	Count	Frequency (%)
True	870275	88.1%
False	113073	11.5%
(Missing)	3921	0.4%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

First rows

	Unnamed: 0	row_id	timestamp	user_id	content_id	task_container_id	user_answer	answered_correctly	prior_question_elapsed_time	prior_question_had_explanation
0	90286654	90286654	0	1917429896	7900	0	0	1	NaN	NaN
1	90286655	90286655	26511	1917429896	7876	1	2	0	21000.0	False
2	90286656	90286656	55312	1917429896	175	2	0	0	21000.0	False
3	90286657	90286657	102436	1917429896	1278	3	0	0	23000.0	False
4	90286658	90286658	188454	1917429896	2063	4	3	0	45000.0	False
5	90286659	90286659	188454	1917429896	2065	4	2	1	45000.0	False
6	90286660	90286660	188454	1917429896	2064	4	2	0	45000.0	False
7	90286661	90286661	264296	1917429896	3365	5	0	0	26666.0	False
8	90286662	90286662	264296	1917429896	3364	5	1	1	26666.0	False
9	90286663	90286663	264296	1917429896	3363	5	2	0	26666.0	False

Last rows

	Unnamed: 0	row_id	timestamp	user_id	content_id	task_container_id	user_answer	answered_correctly	prior_question_elapsed_time	prior_question_had_explanation
987259	47484288	47484288	648640	1003849974	6173	12	0	0	15000.0	False
987260	47484289	47484289	668463	1003849974	6370	13	2	1	29000.0	False
987261	47484290	47484290	1619704	1003849974	6910	14	3	1	17000.0	False
987262	47484291	47484291	1619704	1003849974	6908	14	0	1	17000.0	False
987263	47484292	47484292	1619704	1003849974	6911	14	3	1	17000.0	False
987264	47484293	47484293	1619704	1003849974	6909	14	0	1	17000.0	False
987265	47484294	47484294	1906094	1003849974	7217	15	2	1	68500.0	False
987266	47484295	47484295	1906094	1003849974	7216	15	3	1	68500.0	False
987267	47484296	47484296	1906094	1003849974	7219	15	2	1	68500.0	False
987268	47484297	47484297	1906094	1003849974	7218	15	2	1	68500.0	False

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Missing values

Sample

First rows

Last rows